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Between 1981 and 1984, the average cut 
in the funding of UK universities by the 
University Grants Committee was nearly 
15% in real terms. In retrospect, this 
severe pruning of the university sector in 
the early 1980s has turned out to be a 
prelude to further radical changes in uni¬ 
versity funding. Government policy 
towards universities (and towards higher 
education more generally) is currently be¬ 
ing redesigned to make the funding pro¬ 
cess more competitive in the hope that this 
will increase the efficiency and effective¬ 
ness of higher education in Britain. 

The Government’s policy towards 
higher education has been evolving at a 
rapid pace since 1985 when the Green 
Paper on The Development of Higher 
Education into the 1990s argued that the 
higher education sector was not perform¬ 
ing its job satisfactorily. 2 According to 
the Green Paper, the higher education 
sector could improve its efficiency and 
effectiveness in several ways. The main 
recommendations were as follows: 

!. Higher education should be more 
responsive to the needs of the 
economy. This will require closer links 
to be forged between higher education 
and industry. In addition, it will also 
be necessary to switch the subject mix 
away from the arts and humanities 
towards technical and vocationally- 
related courses. 

2. Higher education depends far too 
heavily on public sector funds and 
greater efforts are needed to raise 
private funds through joint research, 
consultancy and continuing education. 

3. Greater selectivity is needed in the 
allocation of research funding so that 
more resources are concentrated in 
centres of excellence. 

4. The higher education sector should be 
more cost-conscious and should 
manage its resources more efficiently 
and more effectively. This will require 
the construction and regular publica¬ 
tion of a range of performance in¬ 
dicators, These will be used to aid the 
resource allocation process both within 
and between institutions. 

The same broad themes were reiterated 
in the recent White Paper on Higher 
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Education: Meeting the Challenge. 2 In ad¬ 
dition, however, two major changes to the 
organisation and funding of the university 
sector were announced. Firstly, the 
University Grants Committee is soon to 
be replaced by the Universities Funding 
Council. The members of the council will 
be appointed by the Secretary of State for 
Education and Science and will be drawn 
from the academic and non-academic 
world in about equal proportion. Second¬ 
ly, the system of allocating funds to uni¬ 
versities is to be changed from the block 
grant system to one based upon contrac¬ 
tual agreements between the universities 
and the Universities Funding Council. 
Universities will have to offer a clearly 
specified range of educational services in 
exchange for government funding. The 
system of contracting has not yet been 
worked out in detail but the Govern¬ 
ment’s first thoughts are spelt out in a 
consultative document recently produced 
by the Department of Education and 
Science. 4 (1987c). It should also be noted 
that the Government intends to maintain 
tight control over the total level of expen¬ 
diture on higher education and will also 
provide guidelines (to the Universities 
Funding Council and to the universities 
themselves) about the direction in which it 
thinks the universities ought to be going. 

A common theme running through 
both the White Paper and the consultative 
document on contracts is the Govern¬ 
ment’s emphasis on the need for perfor¬ 
mance indicators: 

Arrangements for the flow oj manage¬ 
ment information and for account¬ 
ability from the universities to the Uni¬ 
versities Funding Council and onwards 
to the Government should be much im¬ 
proved. 5 

A key element in the move to a system 
of contracting will be the introduction 
of more systematic review and monitor¬ 
ing of what institutions achieve with 
public funding from the Universities 
Funding Council and the Polytechnics 
and Colleges Funding Council. This 
will require the timely collection, 
analysis and, desirably, publication of 
more information about performance. 
The funding bodies will need at an early 
stage to enter into a dialogue with in- 
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stitutions about what measures of 
performance it might be serviceable and 
feasible to collect, and also how these 
indicators might be used to assess in¬ 
stitutions’ delivery of provision con¬ 
tracted for with public funds . 6 
The Government has gone further than 
simply stating that performance indi¬ 
cators will be needed. It makes it clear 
how it thinks the performance of univer¬ 
sities should be measured: 

Academic standards and the quality of 
teaching in higher education need to be 
judged by reference mainly to students’ 
achievements. The numbers and class 
distribution of degrees awarded provide 
some measure as, conversely, do non¬ 
completion rates. External examiners’ 
reports offer a vital commentary, and 
effective scrutiny of these by institu¬ 
tions is essential. 1 

The subsequent employment patterns 
of students provide some indication of 
the value of higher education courses to 
working life. Evaluation of institu¬ 
tional performance also requires 
students’ achievements to be set 
alongside their entry standards. Greater 
attention needs to be given to these 
questions both nationally and by in¬ 
stitution; and the essential data on 
performance in each institution should 
be published so that its record can be 
evaluated by the funding agencies, 
governing bodies, students and 
employers . 8 

Publication of performance 
indicators 

The reaction of the universities to the 
Government’s demand for regular 
publication of a set of performance in¬ 
dicators has been very quick: the first 
batch of indicators will be published by a 
joint working group of the University 
Grants Committee and the Committee of 
Vice-Chancellors and Principals in the 
autumn of 1987. This first set of data is 
likely to be the forerunner of a regular 
(annual) publication which will ultimately 
provide information on a wide range of 
university activities by cost centre (i.e. 
groups of departments) and by 
institution. 



Qualifications of new entrants 
Undergraduate applications/places ratio 
Funding from consultancy activities (as % 
Private funding of consultancy and res 


F total funding) 

(as % of total funding) 


Staff/student ratios 

Expenditure per student on student services (careers, counselling, ei 

Total costs per student 

Central government revenue per student 

Teaching costs per student 

Administrative costs per student 

Library costs per student 

Computing costs per student 

Ratio of support staff to academic staff 

Teaching indicators 

Staff workloads (teaching contact hours) 

External reviews of courses, teaching methods, examination papers 

Class distribution of degrees awarded 

Non-completion rates 

Success rate of Masters and PhD students 

Student assessment of teaching methods and staff 

First destinations of new graduates 

Careers of graduates five years after graduation 

Research indicators 

Publications (per capita) 

Citations (per capita) 

Research income (as percent of total income) 

Research grants (per capita) 

Peer group assessment of research output 


If the Government’s recommendations 
are followed up, the list of published in¬ 
dicators will ultimately include a wide 
range of variables covering inputs and 
operational activities as well as various in¬ 
dicators of research and teaching outputs. 
An abbreviated list of some possible in¬ 
dicators (gleaned from various sources) is 
given in Table 1. 


“Evaluation involves far 
more than simply providing 
information about a set of 
variables which relate to the 
operation of institutions,” 


Evaluation of performance 
indicators 

Exactly how these indicators are to be 
used, and whether they are likely to be 
helpful in comparing the performance of 
universities (or departments within uni¬ 
versities) is a question which has still not 
been satisfactorily answered. 9 Evaluation 
involves far more than simply providing 
information about a set of variables 
which relate to the operation of institu¬ 
tions. Ideally, objectives should first be 
dearly specified so that an attempt can be 
made to measure the extent to which these 
objectives have been (or are being) achiev¬ 
ed — and at what cost. In practice, it is 
likely that far cruder approaches will be 
adopted in the immediate future. Initially, 
the evaluation process is likely to be based 
on inter-university comparisons of in¬ 
dividual variables such as the employment 
rate of new graduates. The success of a 
university’s graduates in obtaining perma¬ 
nent employment, for example, could be 
used as an indicator of the extent to which 
each institution produces employable 
graduates. Universities with a high pro¬ 
portion of graduates obtaining a perma¬ 
nent job may be judged to have been more 
successful than universities with a low 
proportion of graduates obtaining a per¬ 
manent job. 

It is easy to show that such simplistic 
approaches to measuring the performance 
of universities are likely to be worthless, 
For example, it is possible for a university 
with a high proportion of graduates ob¬ 
taining a permanent job to be performing 
less well than a university with a low pro¬ 
portion. Consider the following example: 
university A has a high proportion of 
graduates obtaining a permanent job 
(within six months of graduating) whereas 
university B has a low proportion. There 
may be very good reasons, however, w ! hy 
A’s graduates get jobs more quickly than 
B’s graduates. University A may have a 
subject mix which is heavily weighted in 


favour of vocationally-related subjects 
whereas university B may have a higher 
proportion of graduates in the arts and 
humanities. Since those graduating in 
vocationally-related subjects (such as 
accountancy) are more likely to know 
what type of job they want than arts and 
humanities students, their search for a job 
is likely to yield quicker results than those 
less firmly committed to a specific career. 
Indeed, many graduating in vocationally- 
related subjects will have a job offer on 
hand even before they graduate. 

Comparisons between universities in 
their graduate employment rates are 
therefore worthless unless differences in 
subject mix are taken into account. This 
can be done by first calculating the 
graduate employment rate that each 
university would have had if its graduates 
had experienced the national employment 
rate in their degree subject. The national 
employment rate in each subject is 
therefore applied to each university’s sub¬ 
ject mix to provide a standardised (or ex¬ 
pected) employment rate for each univer¬ 
sity. This can be used as the yardstick 
against which each university’s actual 
graduate employment rate should be 
compared. 10 

But even this modified indicator may be 
inadequate since differences in the subject 
mix of universities may be only one factor 


amongst many which lead to differences 
in the graduate employment rate between 
nniversities. Other factors which may in¬ 
fluence the success of a university’s 
graduates in the job market include: the 
degree results obtained by a university’s 
graduates; the socio-economic group to 
which the parents of graduates belong; 
the location of the university in relation to 
the geographical distribution of jobs; the 
efficiency of the university’s careers ad¬ 
visory services; and the links between the 
university’s staff and potential employers 
(e.g. through research and consultancy). 
Direct comparisons between universities 
in the proportion of graduates who are 
successful in the job market are therefore 
likely to be of little value since like is not 
being compared with like. 

Another indicator which has been sug¬ 
gested by the Government as a suitable 
measure of performance is the class 
distribution of degrees. 11 The proportion 
of each university’s graduates obtaining a 
first or upper second class honours 
degree, for example, could be used as a 
measure of the quality of graduates pro¬ 
duced by each university. But such a 
measure would be useless per se for 
measuring the quality of teaching in each 
institution .since the quality of inputs (e.g. 
the ability of students) varies between 
universities. 
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Comparisons of degree results between 
universities are therefore unlikely to be of 
much help in measuring the teaching 
performance of universities since the raw 
material inputs may vary in quality 
considerably between institutions. 
Ideally, we need to devise a measure of 
the value added to the knowledge and 
skills of each university’s graduates. An 
obvious approach to constructing a 
measure of value added is to estimate the 
extent to which differences in degree 
results between universities are affected 
by the ability of each university’s students 
(as reflected for example by A-level 
scores). Recent research indicates a very 
strong statistical relationship between the 
mean A-level score of each institution’s 
intake of students and the proportion of 
these students who obtain a first or upper 
second. 12 Having estimated the statistical 
relationship between A-level score and 
degree result, it is possible to construct a 
performance indicator which takes dif¬ 
ferences in the A-level score of each uni¬ 
versity’s students into account when com¬ 
paring degree results. This can be done by 
first calculating the degree results for each 
institution which would have been ex¬ 
pected given the mean A-level score of its 
students and then comparing the actual 
degree results with the expected degree 
results. 

It would be highly dangerous, however, 
to place much faith in such an indicator 
since other factors in addition to the abili¬ 
ty of students may be expected to have an 
effect on degree results. These other fac¬ 
tors would have to be taken into account 
in any estimate oT the value added by 
universities to the knowledge and skills of 
their students. Johnes and Taylor for ex¬ 
ample have found that the ex-Colleges of 
Advanced Technology and the new green¬ 
field universities established in the 1960s 
award a significantly higher proportion of 
upper seconds (on average) than the older 
civic universities. 13 Whether these dif¬ 
ferences are due to corresponding dif¬ 
ferences in the quality of teaching bet¬ 
ween the newer and the older universities 
or whether they are dne to differences in 
the methods of assessing students is 
impossible to say (at this stage). It would 
therefore be folly to use degree results as a 
performance indicator (even allowing for 
differences in the quality of student in¬ 
puts) until more research has been done 
on identifying the factors causing dif¬ 
ferences in degree results between 
universities. 
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A final example of the difficulties in¬ 
volved in devising operationally useful 
performance indicators is provided by 
Bentham, who constructs several in¬ 
dicators relating to the research activity of 
UK geography departments. 14 Bentham’s 
indicators are based on three variables: 
publications, citations and research 
grants. He shows that the ranking of 
departments depends critically on the 
variable selected to measure research 
performance. Only a low correlation was 
found, for example, between publications 
per capita and research grants per capita. 

The low correlation between publica¬ 
tions per capita and research grants per 
capita raises an interesting question since 
it is believed that research income was an 
important input into the UGC’s evalua¬ 
tion of geography departments in its re¬ 
cent (1986) assessment of the research 
strengths of each university. Further¬ 
more, Bentham shows that the UGC’s 
ratings of UK geography departments are 
most closely related to total rather than 
per capita research grant income, in¬ 
dicating a bias against smaller depart¬ 
ments and departments geared to less ex¬ 
pensive types of research activity. The low 
correlation between alternative perfor¬ 
mance indicators and the lack of corres¬ 
pondence between the UGC’s evaluation 
of research strengths and individual in¬ 
dicators (particularly publications per 
capita) is a cause for concern not only 
about the validity of the UGC’s ratings 
but also about the validity of research per¬ 
formance indicators more generally. 

The need for caution in using 
performance indicators 

The examples of performance in¬ 
dicators given in the previous section have 
already indicated that it will not be easy to 
construct meaningful and useful perfor¬ 
mance indicators for the higher education 
sector. It should also be realised that the 
use (and mis-use) of such indicators may 
have several damaging effects on higher 
education. 

Firstly, there will be a strong tempta¬ 
tion to construct indicators from the most 
readily available data (such as the first 
destinations record and the class distribu¬ 
tion of degrees) even though such in¬ 
dicators may provide little useful infor¬ 
mation about the performance of univer¬ 
sities. The first destination of graduates, 
for example, may provide very little in¬ 
dication about the social or economic 
value of different degree subjects but this 
is unlikely to deter the use of such infor¬ 
mation for strategic planning purposes 
unless more relevant data are produced. 15 
More fundamental problems arise in 
assessing the value of outputs such as 
culture, social and political awareness, 
and the acquiring of analytical and critical 
skills. The fact that these types of output 
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cannot be readily measured could result in 
a serious under-valuation of the benefits 
flowing from them. 


“Concentration on the short 
term Is an inevitable 
consequence of the financial 
pressures likely to be imposed 
on the university sector.” 


A second problem with performance in¬ 
dicators is that there will be a tendency for 
higher education institutions to sacrifice 
long-term benefits for more immediate 
short-term benefits in order to improve 
their short-term performance. Concentra¬ 
tion on the short-term is an inevitable 
consequence of the financial pressures 
likely to be imposed on the university sec¬ 
tor. The consultative document on con¬ 
tracts, for example, suggests that such 
contracts should be based upon a three- 
year plan: quick results will therefore be 
needed if universities are to be seen to be 
performing well. This emphasis on the 
short term does not augur well for pro¬ 
gress in basic research. The new policy 
towards assessing the performance of uni¬ 
versities may consequently have detri¬ 
mental effects on the quality of research 
output. Pressure to produce research 
results quickly may induce researchers to 
take fewer risks and to undertake research 
work which is more likely to yield quick, 
but trivial, results. More academics may 
also turn to income-generating project 
work and consultancy in preference to 
tackling fundamental research problems. 

A further adverse consequence of a 
shift away from basic research is an in¬ 
crease in the emigration rate of Britain’s 
best researchers. The brain drain to the 
USA will be reinforced. This will have 
damaging effects on the UK university 
sector which will stretch far into the 
future. 

Conclusion 

Since the severe cut-backs in university 
expenditure in the early 1980s, the 
Government has made it crystal clear that 
the university sector will be expected to 
pay more attention to its efficiency and its 
effectiveness. Universities can therefore 
expect far greater public scrutiny of their 
activities than has traditionally been the 
case. 

In addition to this greater scrutiny of 
their activities, the Government has made 
it clear that the traditional methods of 
allocating resources in the higher educa¬ 
tion sector are to be replaced by a more 
competitive approach. This is reflected in 
the proposed replacement of the Univer¬ 
sity Grants Committee by a Universities 
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Funding Council . 16 The funds made 
available by the Government for the 
university sector will be allocated through 
a system of contracting out educational 
services to individual institutions . 17 These 
contracts will be based upon detailed 
strategic plans drawn up by each institu¬ 
tion and the distribution of the available 
funds will be determined by the relative 
attractiveness of these strategic plans to 
the Funding Council . 16 

A direct implication of the change in 
the method of funding universities from a 
grant system to a contractual system is the 
introduction of a more detailed method of 
appraising the performance of institu¬ 
tions. This will require the construction of 
indicators which can be used for measur¬ 
ing the performance of each institution in 
relation to other institutions in the higher 


education sector. Those institutions which 
perform the best will presumably find it 
easier to attract future funding from the 
Universities Funding Council than those 
institutions which perform less well. The 
Government has already indicated in 
general terms what types of indicators are 
required. Indeed, the universities have 
begun the process of producing and 
publishing a set: of indicators. Precious 
little thought has yet been given, however, 
to the question of how the performance of 
universities (and their constituent parts) 
ought to be measured. The priority has 
been to produce a set of indicators as 
quickly as possible rather than to develop 
an acceptable methodology for evaluating 
the performance of universities. More 
thought now needs to be applied to the 
latter question. 

The main conclusion of this article is 


that great care will be needed in inter¬ 
preting the various indicators which are 
currently being constructed. If these in¬ 
dicators are not used cautiously, major 
mistakes could easily be made in 
allocating resources both between and 
within universities. This is not to argue 
that attempts to measure the performance 
of universities should not be made. In¬ 
deed, universities (like all publicly-funded 
institutions) should welcome the oppor¬ 
tunity to demonstrate their efficiency and 
effectiveness more openly to the taxpayer. 

It would be prudent at this stage to 
label ail so-failed performance indicators 
with the following warning: 

“These indicators may damage the 
health of higher education. Use with ex¬ 
treme caution. Do not take them at face 
value. ” 
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